Enhanced on-object context menus

ABSTRACT

Aspects of the present invention provide context menus useful in, e.g., a computing device receiving user input via a stylus. Icons representing actions performable on an object are displayed in a context menu for the object. Additional aspects of the invention include cascading menus that also minimize hand and/or wrist motion, as well as placement of menus based on user handedness and/or stylus orientation.

FIELD OF THE INVENTION

[0001] This invention relates to an enhanced user interface forperforming actions regarding information displayed on the screen of acomputer or other electronic device. In particular, this inventionrelates to methods, systems and user interfaces for usingcontext-specific menus to perform actions pertaining to a particularobject having a corresponding screen representation.

BACKGROUND

[0002] Computers (including lesser electronic devices with limitedcomputational abilities) are now part of everyday life. Until recently,however, the tasks for which computers could practically be used wassometimes limited by the manner in which a user could input information.Although there are numerous devices with which a computer user canprovide input to a computer, many of these devices (e.g., keyboards,mice, trackballs, touchpads, etc.) have limitations and are notwell-suited for some circumstances. Note-taking is but one example. Manyindividuals lack the typing skills to contemporaneously take notes bytyping on a keyboard. Moreover, many persons find it difficult tosimultaneously use a keyboard and devote attention to other matters,such as participation in a meeting. Conversely, most people can takehandwritten notes while participating in such activities, and can do sorather quickly. Many people also take notes by combining writtenlanguage with drawings, sketches, diagrams, symbols, etc. Keyboards,mice, touchpads, trackballs and various other traditional user inputdevices are not conducive to such activity. Moreover, keyboards, miceand trackballs generally require a surface upon which to rest the inputdevice (and usually the computer), and are thus ill-suited to situationswhere a user may be standing or frequently moving from place to place.

[0003] An input device and technique that addresses these and otherproblems mimics traditional pencil (or pen) and paper. Specifically,certain devices allow a user to create images by moving a stylus acrossa display screen. Using the stylus (or a “pen”) as a writing instrument,the user creates a graphical image of handwriting, symbols, drawingfigures, or anything else that can be drawn with pencil and paper. Theelectronic ink created with the stylus may be maintained or stored inthe graphical form in which it is created, such as a drawing,handwritten notes, etc. The ink may also be converted to another form.For example, various application software is able to recognize orinterpret the electronically handwritten letters, words, etc. andsubstitute text for the electronic “ink.” These devices and associatedrecognition software offer the convenience and intuitive nature ofpencil and paper combined with electronic storage, editing, copying andother advantages of a computer.

[0004] As with other input schemes, however, a user of a pen-basedcomputer will need to execute various commands, make menu choices, andperform numerous other activities beyond the simple creation ofhandwriting or drawing figures. For example, and similar to input usinga mouse or keyboard, a user may need to specify font, color, type,paragraph style, or other text characteristics; may need to specify apredetermined drawing shape such as a line, rectangle, circle, arrow,flow chart box, or other frequently used shape; may need to save anobject on the display into a file; may need to open a new file; may needto cut or copy a portion of a displayed object, or paste somethingpreviously cut or copied; may need to highlight or erase a portion of ascreen object; or may need to perform numerous other commands oractions. In many graphical user interfaces, icons or drop-down textualmenus are provided at the top of the screen, at the side, or elsewhereon a screen. To execute one of the commands corresponding to an icon ortextual menu item, the user typically positions a curser over the iconor menu item, and signals a choice (by, e.g., pressing a mouse or penbutton).

[0005] Another format for presentation of possible commands is thecontext menu. Context menus are described in U.S. Pat. No. 5,664,133.Instead of moving the cursor to a main toolbar or menu, a user positionsthe cursor on (or near) an on-screen object (e.g., a text word,paragraph or other text block; a drawing figure; an image; a controlinterface; etc.) and indicates that a set of menu items specific to thatobject is desired. This indication can occur via the user pressing aparticular button on a pointing device (e.g., pressing a right mousebutton instead of a left mouse button). A context menu is then madeavailable to the user that contains commands and actions (or shortcutsto commands and actions) that pertain to the object in question. Thecontext menu may be displayed near the object in question.

[0006] Context menus can potentially solve certain problems associatedwith pen-based computers. When used for pointing, and unlike a mouse(which is a relative pointer), a stylus is an absolute pointer. A useris therefore often required to move his or her entire arm to navigateacross a screen, not just the fingers or wrist (as is the case with amouse). A stylus may thus require more movement than a mouse to achievethe same distance on the screen, resulting in faster hand and arm musclefatigue. Context menus offer a way of minimizing required stylus traveland reducing fatigue.

[0007] Until now, however, the potential benefits of context menus havebeen undercut by differences between pointing with a stylus and pointingwith a mouse. For example, it is generally more difficult to target ascreen object with a stylus-moved cursor than with a mouse-moved cursor.A mouse typically moves across a desktop or other stable work surface,and the user's hand and arm are often resting on the same work surface.It is therefore easy to make small, controlled cursor movements.Conversely, a user typically holds a stylus above a tablet with lesssupport and stability, making small controlled movements more difficult.Moreover, selecting a screen object by clicking with a mouse button isgenerally a pure vertical movement with one finger; the user can easilyseparate button-clicking finger movements from cursor-moving handmovements. With a stylus, however, a user may use the same arm, wrist,hand, and finger muscles to both move the cursor (by moving the stylustip) and select a screen object (by, e.g., maintaining the cursor in alocation or making a gesture). Even if a separate button is provided onthe stylus, use of a stylus button is likely to cause the stylus tip,and thus the cursor, to move slightly. A stylus-controlled cursor mayalso be more difficult to see than a mouse-controlled cursor. Unlike amouse, a stylus typically requires a user to rest his or her hand on thedisplay screen, potentially obscuring the cursor location.

[0008] To overcome the targeting and visualization difficulties of astylus, it is often desirable to increase the target size for a screenobject. By way of hypothetical example, a screen display for particularobject may have a target size for mouse-controlled cursor selection of 5pixels by 5 pixels. To facilitate use of a stylus input device, thetarget size of that object might be increased to 10 pixels by 10 pixels.In other words, selecting the object in an environment designed for amouse-controlled cursor requires placement of the cursor into a 5×5square, while selection in an environment designed for stylus selectiononly requires placement of the cursor in a 10×10 area.

[0009] However, increasing object selection target size conflicts withthe goal of minimizing stylus movements across the screen. As targetsbecome larger, more screen space is needed to display multiple targets,and more distance must be traversed across the screen. With regard tocontext menus, the number of menu selections that can be presented in acontext menu is thereby limited. Using conventional context menus withmenu choices sized for stylus selection, studies have shown that morethan seven menu items requires undesirable hand or wrist motion.

[0010] Accordingly, there remains a need for systems, methods and userinterfaces that provide improved context menus for pen-based user input.

SUMMARY

[0011] Aspects of the present invention provide solutions to one or moreof the problems highlighted above. Aspects also relate to methods andsystems for selecting actions that can be performed with regard to anobject having an associated representation on the display screen of acomputer. Among other features, aspects of the invention allow displayof icons, representing actions performable on an object, in a contextmenu for the object. By presenting context menu choices in the form oficons, it is possible to minimize hand and/or wrist motion by a usermaking menu selections with a stylus user input device. Additionalaspects of the invention include cascading levels of context menus thatalso minimize hand and/or wrist motion, as well as placement of menusbased on user handedness and/or stylus orientation. Further advantagesand features of the invention are set forth in the detailed description,or will be apparent to persons skilled in the art in light of thatdescription.

BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The foregoing summary is better understood when read inconjunction with the accompanying drawings, which are included by way ofexample, and not by way of limitation with regard to the claimedinvention.

[0013]FIG. 1 shows a schematic diagram of a general-purpose digitalcomputing environment that can be used to implement various aspects ofthe invention.

[0014]FIG. 2 shows a plan view of a tablet computer and stylus that canbe used in accordance with various aspects of the present invention.

[0015]FIG. 3 shows a context menu according to one aspect of theinvention.

[0016] FIGS. 4A-4D show the advantages of the context menu of FIG. 3over other possible solutions.

[0017]FIG. 5 shows movement of a context menu according to anotheraspect of the invention.

[0018] FIGS. 6A-6I show cascading levels of context menus according toanother aspect of the invention.

[0019]FIGS. 7A & 7B show placement of a context menu according toanother aspect of the invention.

[0020]FIG. 8 shows placement of a context menu according to anotheraspect of the invention.

[0021]FIG. 9 shows a non-rectilinear context menu according to anotheraspect of the invention.

DETAILED DESCRIPTION

[0022] This invention relates to methods, systems and user interfacesfor using context-specific menus to perform actions pertaining to aparticular object having a corresponding screen representation. Thefollowing description is divided into various subsections: Definitions;Illustrative Operating Environment; Enhanced On-Object Context Menus;Additional Aspects; and Conclusion. These subsections are for thereader's convenience and not intended as limitations upon the invention.

[0023] Definitions

[0024] Unless otherwise indicated, various terms as used in thisspecification and in the claims have the following meanings:

[0025] Action: A modification of, or a command, function, or otheractivity relating to, an object. An action includes, inter alia,creating an object.

[0026] Icon: A small on-screen pictorial representation. Although iconsare primarily non-textual, they may include letters or numbers.

[0027] (Electronic) Ink: A sequence or set of points, displayed ordisplayable on a CRT, LCD, or other display device, which were createdby moving a stylus, mouse or other input device so as to mimic penciland paper or other types of non-electronic writing or drawing. Theelectronic ink may include additional properties and methods resultingin a rich representation of the ink. Also, the ink may be stored as anink object which may include properties and/or methods.

[0028] Object: As commonly used in the art, an object includescombinations of computer code and data used to represent quantities orconcepts with which a user can interact. An object may have variablesthat can have values, and may also have associated procedures, methodsand other functions.

[0029] Text: Letters, words, sentences, paragraphs, etc. havingcharacters that can be represented by unique codes (e.g., ASCII,Unicode, etc.). Text also includes non-letter characters and symbols.

[0030] Illustrative Operating Environment

[0031] Aspects of the invention may be implemented with instructionsthat can be executed on a computing device. Accordingly, an illustrativecomputing environment for executing such instructions will be described.Aspects of the invention may be described in the general context ofcomputer-executable instructions, such as program modules, executed byone or more computing devices. Generally, program modules includeroutines, programs, objects, components, data structures, etc. thatperform particular tasks or implement particular abstract data types.Typically, the functionality of the program modules may be combined ordistributed as desired. Because the invention may be implemented usingsoftware, it may useful to briefly discuss the components and operationof a typical programmable computer on which various aspects of theinvention may be employed. One illustrative computer system isillustrated in FIG. 1. The system includes a general-purpose computer100. This computer 100 may take the form of a personal digitalassistant; a tablet, desktop or laptop personal computer; a networkserver, or the like. Computer 100 typically includes at least some formof computer readable media. Computer readable media can be any availablemedia that can be accessed by the computer 100. By way of example andnot limitation, computer readable media may comprise computer storagemedia and communication media. Computer storage media includes volatileand nonvolatile, removable and non-removable media implemented in anymethod or technology for storage of information such as computerreadable instructions, data structures, program modules or other data.Computer storage media includes, but is not limited to, RAM, ROM,EEPROM, flash memory or other memory technology, CD-ROM, digitalversatile disks (DVD) or other optical storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium which can be used to store the desired informationand which can be accessed by the computer 100 using appropriate hardwareand interfaces. Communication media typically embodies computer readableinstructions, data structures, program modules or other data in amodulated data signal such as a carrier wave or other transportmechanism and includes any information delivery media. The term“modulated data signal” means a signal that has one or more of itscharacteristics set or changed in such a manner as to encode informationin the signal. By way of example, and not limitation, communicationmedia includes wired media such as a wired network or direct-wiredconnection, and wireless media such as acoustic, RF, infrared and otherwireless media. Combinations of any of the above should also be includedwithin the scope of computer readable media.

[0032] With reference to FIG. 1, computer 100 includes a processing unit110, a system memory 120, and a system bus 130 that couples varioussystem components including the system memory to the processing unit110. The system bus 130 may be any of several types of bus structuresincluding a memory bus or memory controller, a peripheral bus, and alocal bus using any of a variety of bus architectures. The system memory120 may include read only memory (ROM) 140 and random access memory(RAM) 150.

[0033] A basic input/output system 160 (BIOS), containing the basicroutines that help to transfer information between elements within thecomputer 100, such as during start-up, may be stored in the ROM 140. Thecomputer 100 may also include any or all of a hard disk drive 170 forreading from and writing to a hard disk (not shown), a magnetic diskdrive 180 for reading from or writing to a removable magnetic disk 190,and an optical disk drive 191 for reading from or writing to a removableoptical disk 192 such as a CD ROM, DVD or other optical media. The harddisk drive 170, magnetic disk drive 180, and optical disk drive 191 maybe connected to the system bus 130 by a hard disk drive interface 192, amagnetic disk drive interface 193, and an optical disk drive interface194, respectively. The drives and their associated computer-readablemedia provide nonvolatile storage of computer readable instructions,data structures, program modules and other data for the computer 100. Itwill be appreciated by those skilled in the art that other types ofcomputer readable media that can store data that is accessible by acomputer, such as magnetic cassettes, flash memory cards, digital videodisks, Bernoulli cartridges, random access memories (RAMs), read onlymemories (ROMs), and the like, may also be used in the example operatingenvironment. Also, it should be appreciated that more portableembodiments of the computer 100, such as a tablet personal computer orpersonal digital assistant, may omit one or more of the computer storagemedia devices discussed above.

[0034] A number of program modules can be stored on the hard disk drive170, magnetic disk 190, optical disk 192, ROM 140 or RAM 150, includingan operating system 195, one or more application programs 196, otherprogram modules 197, and program data 198. A pen digitizer 165 andaccompanying pen or stylus 166 may be provided in order to digitallycapture freehand input. Although a direct connection between the pendigitizer 165 and the serial port interface 106 is shown, in practice,the pen digitizer 165 may be coupled to the processing unit 110directly, or via a parallel port (or other) interface and the system bus130 as known in the art. Furthermore, although the digitizer 165 isshown apart from the monitor 107, it is preferred that the usable inputarea of the digitizer 165 be co-extensive with the display area of themonitor 107. Further still, the digitizer 165 may be integrated in themonitor 107, or may exist as a separate device overlaying or otherwiseappended to the monitor 107. A user might also enter commands andinformation into the computer 100 through input devices such as akeyboard 101 and pointing device 102. Other input devices (not shown)may include a microphone, joystick, game pad, satellite dish, scanner orthe like. These and other input devices are often connected to theprocessing unit 110 through a serial port interface 106 that is coupledto the system bus, but may be connected by other interfaces, such as aparallel port, game port or a universal serial bus (USB). Further still,these devices may be coupled directly to the system bus 130 via anappropriate interface (not shown). Monitor 107 or other type of displaydevice is also connected to the system bus 130 via an interface, such asa video adapter 108. In addition to the monitor, computers may includeother peripheral output devices (not shown), such as speakers andprinters.

[0035] The computer 100 may also operate in a networked environmentusing logical connections to one or more remote computers, such as aremote computer 109. The remote computer 109 can be a server, a router,a network PC, a peer device or other common network node, and mayinclude many or all of the elements described above relative to thecomputer 100, although only a memory storage device 111 has beenillustrated in FIG. 1. The possible logical connections depicted in FIG.1 include a local area network (LAN) 112 and a wide area network (WAN)113. Such networking environments are commonplace in offices,enterprise-wide computer networks, intranets and the Internet.

[0036] When used in a LAN networking environment, the computer 100 maybe connected to the local network 112 through a network interface oradapter 114. When used in a WAN networking environment, the computer 100typically includes a modem 115 or other means for establishing acommunications over the wide area network 113, such as the Internet. Themodem 115, which may be internal or external, is connected to the systembus 130 via the serial port interface 106. In a networked environment,program modules depicted relative to the computer 100, or portionsthereof, may be stored in the remote memory storage device.

[0037] It will be appreciated that the network connections shown areexemplary and other techniques for establishing a communications linkbetween the computers can be used. The existence of any of variouswell-known protocols such as TCP/IP, Ethernet, FTP, HTTP and the like ispresumed, and the system may be operated in a client-serverconfiguration to permit a user to retrieve web pages from a web-basedserver. Any of various conventional web browsers can be used to displayand manipulate data on web pages.

[0038]FIG. 2 illustrates a tablet PC 201 that can be used in accordancewith various aspects of the present invention. Any or all of thefeatures, subsystems, and functions in the system of FIG. 1 can beincluded in the computer of FIG. 2. Tablet PC 201 includes a largedisplay surface 202, e.g., a digitizing flat panel display, preferably,a liquid crystal display (LCD) screen, on which a plurality of windows203 may be displayed. Using stylus 204, a user can select, highlight,and/or write on the digitizing display surface 202. Examples of suitabledigitizing display surfaces 202 include electromagnetic pen digitizers(such as, e.g., pen digitizers available from Wacom TechnologyCorporation of Vancouver, Wash.). Other types of pen digitizers, e.g.,optical digitizers, may also be used. Tablet PC 201 interprets gesturesmade using stylus 204 in order to manipulate data, enter text, createdrawings, enter system commands and/or execute conventional computerapplication tasks such as spreadsheets, word processing programs, andthe like.

[0039] The stylus 204 may be static, whereby pressure upon (or otherphysical contact with) the display screen or other digitizing surface isnecessary to effect input. Stylus 204 may also be dynamic, with distancefrom and/or orientation with regard to the screen (or other digitizingsurface) being detectable. This could facilitate, e.g., “hovering,”whereby a cursor is located on the screen by holding the stylus over alocation without actually touching the screen or other digitizingsurface. Stylus 204 could also be equipped with one or more buttons orother features to augment its selection capabilities. In one embodiment,the stylus 204 could be implemented as a “pencil” or “pen”, in which oneend constitutes a writing portion and the other end constitutes an“eraser” end, and which, when moved across the display, indicatesportions of the display are to be erased. Stylus 204 may also have oneor more buttons that correspond generally to “left-click,” “right-click”or other buttons often found on a computer mouse; one or more of thesebuttons might be programmable by a user. Other types of input devices,such as a mouse, trackball, or even a keyboard, could also be used withtablet PC 201. Additionally, a user's own finger could be the stylus 204and used for selecting or indicating portions of the displayed image ona touch-sensitive or proximity-sensitive display. Region 205 shows afeedback region or contact region permitting the user to determine wherethe stylus 204 has contacted the display surface 202.

[0040] Enhanced On-Object Context Menus

[0041] Although the invention will primarily be described with regard toa computer configured for user input with a stylus, the invention mayalso be advantageously implemented with regard to other manners of userinput (including mice, trackballs, touch pads, etc.). Accordingly,except where specifically claimed, the invention is not limited tostylus-based user input.

[0042] Using a device such as the tablet PC 201 and stylus 204 shown inFIG. 2, a user may create (or retrieve from memory or another source) anobject by moving the stylus across a display having an incorporateddigitizer. The object could be displayed on the screen as ink, orotherwise have a corresponding graphical (or even textual)representation. The object (and its associated ink or other visualrepresentation) could represent handwriting, which the user may laterconvert to text by way of one or more handwriting recognition engines.The object may be part of a drawing figure which the user is sketching.The object may be a pre-existing text document. The object may be ofsome other type. Regardless of the type of object (or objects) involved,the user may need to perform one or more actions with regard to thatobject (or objects). To do so, the object(s) to be acted upon should beidentified. Such identification could occur in various manners, as areknown in the art. As one alternative, the user might highlight thevisual representation of the object(s) of interest. The user might do soby pressing a particular button on the stylus and moving the stylusacross the visual representation of the object(s). Highlighting mightalso be invoked by first performing a designated stylus gesture. Asanother alternative, the user might use the stylus to create an outlinesurrounding the visual representation of the object(s). As yet anotheralternative, the user may have created the object within an applicationprogram that looks to the most recently modified object when an actionis requested. As a still further alternative, the object's visualrepresentation may have tags or other associated in-document indicators(e.g., a small square, circle or other image located on or near theobject); the user might select the object by tapping on or otherwiseselecting the tag.

[0043] To perform the desired action upon the object, an aspect of thepresent invention allows the user to initiate a context menu for theobject. The context menu may present the user with all or a subset ofall the possible actions applicable to the object. FIG. 3 shows acontext menu according to one embodiment of the invention. The user hascreated an object having associated ink 302, selected the object bydrawing outline 304 or otherwise selecting the object, and theninitiated context menu 306. The user may have initiated context menu 306by performing a certain stylus gesture, by pressing a particular buttonon the stylus (e.g., a button analogous to a “right-click” with amouse), or in some other manner known in the art. Notably, the inventiondoes not require that the user first identify the object to be actedupon and then initiate context menu 306. As one alternative, the usercould first initiate context menu 306 and then be prompted to identifythe object to be acted upon. As another alternative, the user may havepreviously activated context menu 306, and then “pinned” it to theactive document. In such an alternative embodiment, context menu 306might remain at a corner of the display (or other unobtrusive location),and then be “dragged” over ink associated with the object to be actedupon. This embodiment is also shown in FIG. 5, with the menu 306′representing the menu prior to being moved to ink 302.

[0044] Context menu 306 presents the user with various possible actionsrepresented by icons 308-324. These icons may be the same icons that areotherwise familiar to the user from prior use of the program, or fromother programs. By way of example, FIG. 3 shows icons that may beapplicable to one type object in an application program compatible withthe WINDOWS XP operating system available from Microsoft Corporation ofRedmond Wash.: undo previous action (308); redo (310); erase (312); cut(314); copy (316); paste (318); change color (320); change thickness(322); and change highlighting (324). It will be appreciated that theicons shown are merely examples, and other icons could be included inaddition to (or instead of) those shown. As indicated in FIG. 3, contextmenu 306 could also include textual menu choices.

[0045] Aspects of the present invention incorporate icons into a contextmenu to display a larger number of menu choices in a more compact area.In this manner, a user is presented with a larger number of possibleactions, while at the same time minimizing necessary hand or wristmovement. In some embodiments, the icons may be size-optimized forselection with a stylus user input device. In one example, as seen inFIGS. 4A-4D, differences in geometry of icons versus geometry of textallows a larger number of menu selections to be presented in a smallerscreen area. More specifically, icons are more easily contained inregions that are generally uniform in overall size and shape. Moreover,icons can more easily be allocated screen areas that have a non-extremeaspect ratio. In other words, an icon can be allocated to a screenregion with a width that is generally equal to its height, such as asquare or circle. This allows for more efficient use of area within acontext menu, and provides “targets” that are easier for a user toselect with a stylus. For example, FIG. 4A shows context menu 306 incomparison to a context menu 406 having textual menu choices in place ofthe icons in menu 306. The dashed line surrounding each textual menuchoice indicates the target area within which a stylus must be placed toselect that menu option. As is evident, menu 406 is significantlylarger, and would require significantly larger hand/wrist movement tomove though the full range of options. Of course, it is possible toreduce the size of the text (and the corresponding target areas) tocrowd more options into a smaller area. As shown in FIGS. 4B and 4C,however, this approach has its own disadvantages. If the height of eachtextual menu choice is reduced so as to fit into the same area as icons308-324 (as shown by menu 406′ in FIG. 4B), the significantly thinnertarget areas would be more difficult for a user to successfully selectwith a stylus, and would also be more difficult to see. If the width ofeach textual menu choice is reduced so as to allow multiple columns ofmenu options (as shown by menu 406″ in FIG. 4C), the options are muchless readable. As shown with menu 40′″ in FIG. 4D, it is also possibleto variably and moderately reduce both size and width of target areasfor textual menu choices, and then distribute those menu choices in a“patchwork” fashion. However, this is also less easily readable, andcould present difficulties in terms of software coding and performance.Moreover, many menu options would not be satisfactorily expressible inshort words, and thus it might not always be possible to “patchwork”some combinations of options while maintaining a satisfactory minimumtarget size for each option.

[0046] The menu choices (and corresponding icons) presented in a contextmenu can represent those commands or actions most likely to be neededfor the object of interest. The determination of which icons to includecan be made in various manners. As one alternative, some types ofobjects may have standard or default actions/icons displayed. As anotherpossibility, some of the actions/icons could be standard for a givenobject type, with the remainder of the actions/icons determined based onuser selection. For example, the user may initially configure theapplication in which an object is created to provide certain contextmenu choices. As another possibility, some of the context menuactions/icons could be standard for a particular object, with otheractions/icons representing actions the user most recently chose from amain menu or toolbar. Using FIG. 3 as an example, “undo,” “redo,” “cut,”“copy” and “paste” might be standard commands for objects such as thoserepresented by ink 302, with the remaining icons representing commandsthe user has previously chosen from another menu or toolbar. Similarly,the order of displayed icons could vary. As one example, the icons couldbe reordered so as to place the most recently chosen (or most frequentlychosen) actions first. The user might also configure the applicationsoftware to present icons in a certain order.

[0047] Additional Aspects

[0048]FIGS. 6A through 6I illustrate additional aspects of theinvention. As set forth below, hand and wrist motion may also beminimized through the use of cascading levels of context menus. Unlikeconventional cascading menus, which may require a user to traverse anentire series (or tree) of cascaded menus for each menu choice, thepresent invention permits a user to cancel one or more lower levels ofcascaded context menus while leaving one or more higher level contextmenus in place. Cascading context menus may thus be advantageouslyconfigured to allow rapid traversal between sets of menu options. Thesets can be grouped by frequency of use, by user designation, or inanother manner.

[0049]FIG. 6A shows a context menu 506 for ink 302. The user may haveinvoked context menu 506 in various manners, as previously described.Context menu 506 contains four choices identified by text: “Edit”;“Format”; “Undo”; and “Redo.” These menu choices are merely examples forthe purpose of illustrating this aspect of the invention, and other menuchoices representing other actions could be implemented. Similarly, fourchoices are displayed so as not overly obscure the drawing, but more orfewer choices could be presented. As seen in FIG. 6A, two of the choicesin context menu 506 have an indicator 508 showing that additionalchoices are available in a lower level menu. For example, and as shownin FIG. 6B, selecting “Edit” causes display of lower level context menu516. Menu 516 displays four menu choices (or options) which are logicalextensions of the original menu choice (“edit”): “cut,” “copy,” “paste”and “delete.” As also shown in FIG. 6B, the orientation of indicator 508may change to indicate that the menu choice “edit” has been expanded toshow a lower level context menu having related choices.

[0050] Context menu 516 may also be displayed generally in line withcontext menu 506, beginning near the location of the stylus tip (orcursor moved by other pointing device) used to select “edit.” As alsoshown in FIG. 6B, this reduces hand and wrist motion from what would beneeded for conventionally cascaded menus, which typically expand to theside. The solid outline of a stylus labeled “A” represents the positionof the stylus to select “cut.” The dashed outline of the stylus labeled“B” represents the position of the stylus to select “edit.” The dashedoutline of the stylus labeled “C” represents the position of the stylusto select “cut” in a conventionally placed cascading menu. Although theplacement of menu 516 may obscure some choices in one or more higherlevel menus (such as menu 506), the original menu choice (“edit”) fromthe higher level menu is still visible. Moreover, the user may return tothe higher level menu 506 by again tapping or otherwise selecting “edit”or its indicator 508. Upon a second selection of “edit” or its indicator508, menu 516 would disappear, context menu 506 would remain, andindicator 508 would return to its original orientation.

[0051]FIGS. 6C through 6G illustrate multiple levels of cascaded contextmenus and the ease with which a user may traverse between levels ofcontext menus in a cascade. So as not to unnecessarily obscure thedrawing, ink 302 is not shown. FIG. 6C shows context menu 506 uponinitiation by the user. In FIG. 6D, the user then chooses “format,”causing second level context menu 536 to cascade below. Indicator 508 incontext menu 506 has changed orientation to indicate that “format” hasbeen expanded to show a lower level context menu having related choices.As seen in FIG. 6D, context menu 536 contains three choices (or options)which are logical extensions of the original menu choice (“format”):“color” “thickness,” and “highlight.” Each of the choices in menu 536has an indicator 538 showing that additional choices are available inlower level menus. As shown in FIG. 6E, the user has chosen “color” frommenu 536. Indicator 538 has changed orientation, and third level contextmenu 546 has cascaded below. Context menu 546 contains five choiceswhich are logical extensions of the previous menu choice (“color”):“red,” “yellow,” “blue,” “green” and “violet.” As is clear from FIG. 6E,the cascaded context menus have been displayed in a minimal area, buteach of the user's menu choices in the present cascade (“format” and“color”) is still visible.

[0052] The user may the traverse back up through the levels of contextmenus by reselecting a prior menu choice (or its indicator). Forexample, if the user reselected the “format” choice in FIG. 6E (or itsindicator 508), menus 536 and 546 would disappear, but menu 506 wouldremain. The menus could optionally be configured such that, whenever auser makes a menu choice that does not have an associated lower levelmenu, the menu having that choice disappears, but all higher level menusremain. For example, if the user chose “blue” in FIG. 6E, menu 546 mightdisappear, but menu 536 could remain so as to allow further choices frommenu 536.

[0053]FIG. 6F shows what has occurred after the user closed menu 546.Indicator 538 has returned to its original orientation, and all choicesin menu 536 are again visible. As shown in FIG. 6G, the user has nowchosen “thickness” in menu 536. Indicator 538 for “thickness” haschanged orientation, and third level context menu 556 has cascadedbelow. Context menu 556 contains seven choices of line thickness orstyle, which are logical extensions of the previous menu choice(“thickness”). As is clear from FIG. 6G, the cascaded context menus havebeen displayed in a minimal area, but each of the user's menu choices inthe present cascade (“format” and “thickness”) is still visible. Theuser may then make a choice from menu 556, and/or continue to traverseup and down the levels of context menus as many times as may benecessary.

[0054] Although FIGS. 6C-6G only show three levels of cascaded contextmenus, additional levels could be included following the same patterndescribed. Moreover, the use of cascading levels of context menus is notlimited to actions which merely change appearance of a line or otherdrawing figure. As but one possible example, levels of context menuscould be used for mathematical operations. The first level might containchoices for numbers and a choice for a second level menu of mathematicaloperators. The second level menu may have various basic mathematicaloperators (e.g., add, subtract, multiply and divide, etc.), as well asone or more choices for lower level menus of specialized operators. Asbut one example, the second level menu may have a choice called “trig”which, when selected, causes display of a third level menu havingvarious trigonometric functions (e.g., sine, cosine, tangent, etc.). Thethird level menu may also have one or more choices for even lower levelmenus of even more specialized trigonometric functions. As but anotherexample, menu levels of commonly used words can be implemented so as toallow construction of simple or commonly used sentences. A first levelmenu might have choices for nouns and a choice called “verb.” Selecting“verb” could open a second level menu having choices for verbs, and achoice called “object” (note that object is used here in a grammatical,and not a software, sense). Selecting “object” might open a third levelmenu having choices for objects. As but another possible example, aprogram for drawing electrical circuit schematics may have voltage andcurrent sources on level one, resistors on level two, capacitors onlevel three, transistors on level four, etc. Numerous otherpossibilities exist.

[0055] The aspects of the invention illustrated in FIGS. 6A-6G can alsobe incorporated into context menus having icons. As shown in FIG. 6H,selecting icon 314′ may cause display of lower level menu 386. Lowerlevel menu 386 offers choices represented by icons 341-349, and icon 345has an indicator showing that an additional level of choices isavailable. Selection of icon 345 causes display of menu 396, which haschoices represented by icons 441-449. Even further levels of menus couldbe added.

[0056] An additional aspect of the invention accommodates the right- orleft-handedness of the user, and orients the context menu so as tominimize hand and wrist motion. For example, if the user is righthanded, the context menu might be displayed to the left of the styluswhen the context menu is initiated. The large star in FIG. 7A representsthe location of the cursor when the context menu is initiated. Becausethe cursor is typically located near the end of the stylus with whichthe user is pointing to the screen, the cursor provides a referencepoint for a likely location of the user's hand (which is holding thestylus) with respect to the screen, if it is assumed the user is righthanded. In this manner, the context menu can be placed so that the handof a right-handed user does not obscure the context menu. Conversely, ifthe user is left-handed, the context menu might be displayed to theright of the stylus when the context menu is initiated, shown in FIG.7B. The handedness of the user might be set by the user when initiallyconfiguring an application or operating system program; might bedetected based on the orientation of the stylus; or might be establishedin some other manner. Similarly, stylus orientation could be used toadjust context menu positioning in other situations. As illustrated inFIG. 8, the user might be reviewing a spreadsheet, and using the stylusto “check off” cells as they are reviewed. As shown by the dashedoutline of a stylus, the user may be holding the stylus so as tomaximize his or her ability to see a particular row in the spreadsheet.The orientation of the stylus is detected, and the context menugenerated above the selected object.

[0057] In other embodiments, the context menu need not be rectilinear.As shown in FIG. 9, the context menu and icons could be arranged in anarc-like or other fashion so as to be easily reachable by the userwithout moving his or her hand or wrist.

[0058] Conclusion

[0059] Although several examples of carrying out the invention have beendescribed, those skilled in the art will appreciate that there arenumerous variations and permutations of the above described methods,user interfaces and other aspects of the invention that fall within thespirit and scope of the invention as set forth in the appended claims.In addition to the alternatives and variations already discussed,further variations are possible and within the scope of the invention.For example, the regions, buttons and other features of the describedcontext menus can be rearranged, combined or otherwise modified, and/oradditional features added. These and other modifications are within thescope of the invention, which is only to be limited by the attachedclaims.

1. In a computer system having a display screen, a graphical userinterface, and a user interface selection device, a method of selectingactions performable with regard to an object having an associatedrepresentation on the screen, comprising: receiving a user request for adisplay of menu choices corresponding to the actions performable withregard to the object; displaying the menu choices, wherein each actionperformable with regard to the object is identified by a unique icon,the icons are displayed in an array of at least two columns and tworows, and the icons are displayed in a menu which is positioned on thescreen based upon the location of the screen representation of theobject; receiving a user selection of an icon; and performing the actionidentified by the selected icon.
 2. The method of claim 1, whereinreceiving a user request for a display of menu choices comprisesreceiving a request initiated with a stylus user input selection device,and wherein receiving a user selection comprises receiving a selectionmade with the stylus.
 3. The method of claim 2, wherein displaying themenu choices comprises displaying the icons spaced such that any of theicons can be selected without substantial movement of the hand or wristof the user holding the stylus.
 4. The method of claim 3, wherein aportion of the menu is positioned in a display screen location nearestto the position of the stylus tip when the request for a display of menuchoices was received.
 5. The method of claim 3, further comprisingreceiving an indication of the handedness of the user, and whereindisplaying the menu choices comprises displaying the icons in a menupositioned on the screen based upon the indication of the handedness ofthe user.
 6. The method of claim 3, further comprising receiving anindication of the orientation of the stylus with respect to the displayscreen, and wherein displaying the menu choices comprises displaying theicons in a menu positioned on the screen based upon the indication ofthe stylus orientation.
 7. The method of claim 3, further comprising;continuing to display the icons after selection of an icon; receiving anidentification by the user of a second object; receiving a secondselection of an icon; and performing the action identified by the secondselected icon.
 8. The method of claim 7, further comprising: moving thedisplayed icons across the display screen in response to a correspondingmovement of the stylus with respect to the display screen.
 9. The methodof claim 3, further comprising displaying, in response to a userselection of an initial icon, a second level menu of additional icons,wherein receiving a user selection of an icon with the stylus comprisesreceiving a selection of an icon in the second level menu, and whereinperforming the action identified by the selected icon comprisesperforming the action identified by the icon selected from the secondlevel menu.
 10. The method of claim 3, further comprising: displaying,in response to a user selection of an initial icon, a second level menuof additional icons cascaded from the initial icon; receiving aselection of an icon from the second level menu; and displaying, inresponse to selection of the icon from the second level menu, a thirdlevel menu of additional icons cascaded from the second level menu icon,wherein the second and third level menus are cascaded in substantialalignment from the initial icon, receiving a user selection of an iconwith the stylus comprises receiving a selection of an icon in the thirdlevel menu, and performing the action identified by the selected iconcomprises performing the action identified by the icon selected from thethird level menu.
 11. A computer-readable medium havingcomputer-executable instructions for performing steps comprising:receiving a user request for a display of menu choices corresponding toactions performable with regard to an object, wherein the object has arepresentation upon a display screen of a computer system having agraphical user interface; displaying the menu choices, wherein eachaction performable with regard to the object is identified by a uniqueicon, the icons are displayed in an array of at least two columns andtwo rows, and the icons are displayed in a menu which is positioned onthe screen based upon the location of the screen representation of theobject; receiving a user selection of an icon; and performing the actionidentified by the selected icon.
 12. The computer-readable medium ofclaim 11, wherein receiving a user request for a display of menu choicescomprises receiving a request initiated with a stylus user inputselection device, and wherein receiving a user selection comprisesreceiving a selection made with the stylus.
 13. The computer-readablemedium of claim 12, wherein displaying the menu choices comprisesdisplaying the icons spaced such that any of the icons can be selectedwithout substantial movement of the hand or wrist of the user holdingthe stylus.
 14. The computer-readable medium of claim 13, wherein aportion of the menu is positioned in a display screen location nearestto the position of the stylus tip when the request for a display of menuchoices was received.
 15. The computer-readable medium of claim 13,comprising further computer-executable instructions for performing thestep of receiving an indication of the handedness of the user, andwherein displaying the menu choices comprises displaying the icons in amenu positioned on the screen based upon the indication of thehandedness of the user.
 16. The computer-readable medium of claim 13,comprising further computer-executable instructions for performing thestep of receiving an indication of the orientation of the stylus withrespect to the display screen, and wherein displaying the menu choicescomprises displaying the icons in a menu positioned on the screen basedupon the indication of the stylus orientation.
 17. The computer-readablemedium of claim 13, comprising further computer-executable instructionsfor performing the steps of: continuing to display the icons afterselection of an icon; receiving an identification by the user of asecond object; receiving a second selection of an icon; and performingthe action identified by the second selected icon.
 18. Thecomputer-readable medium of claim 17, comprising furthercomputer-executable instructions for performing the step of moving thedisplayed icons across the display screen in response to a correspondingmovement of the stylus with respect to the display screen.
 19. Thecomputer-readable medium of claim 13, comprising furthercomputer-executable instructions for performing the step of displaying,in response to a user selection of an initial icon, a second level menuof additional icons, wherein receiving a user selection of an icon withthe stylus comprises receiving a selection of an icon in the secondlevel menu, and wherein performing the action identified by the selectedicon comprises performing the action identified by the icon selectedfrom the second level menu.
 20. The computer-readable medium of claim13, comprising further computer-executable instructions for performingthe steps of: displaying, in response to a user selection of an initialicon, a second level menu of additional icons cascaded from the initialicon; receiving a selection of an icon from the second level menu; anddisplaying, in response to selection of the icon from the second levelmenu, a third level menu of additional icons cascaded from the secondlevel menu icon, wherein the second and third level menus are cascadedin substantial alignment from the initial icon, receiving a userselection of an icon with the stylus comprises receiving a selection ofan icon in the third level menu, and performing the action identified bythe selected icon comprises performing the action identified by the iconselected from the third level menu.
 21. A computer system, comprising: adisplay screen; a user interface selection device; and a processor incommunication with the display screen and the user interface selectiondevice, and configured to: provide a graphical user interface, receive auser request for a display of menu choices corresponding to actionsperformable with regard to an object, display the menu choices, whereineach action performable on the object is identified by a unique icon,the icons are displayed in an array of at least two columns and tworows, and the icons are displayed in a menu which is positioned on thescreen based upon a location of a screen representation of the object,receive a user selection of an icon, and perform the action identifiedby the selected icon.
 22. The computer system of claim 21, wherein theuser interface selection device comprises a stylus, wherein theprocessor is configured to receive a user request for a display of menuchoices by receiving a request initiated with the stylus, and whereinthe processor is configured to receive a user selection by receiving aselection made with the stylus.
 23. The computer system of claim 22,wherein the processor is configured to display the menu choices bydisplaying the icons spaced such that any of the icons can be selectedwithout substantial movement of the hand or wrist of the user holdingthe stylus.
 24. The computer system of claim 23, wherein the processoris configured to position a portion of the menu in a display screenlocation nearest to the position of the stylus tip when the request fora display of menu choices is received.
 25. The computer system of claim23, wherein the processor is configured to receive an indication of thehandedness of the user, and wherein the processor is configured todisplay the menu choices by displaying the icons in a menu positioned onthe screen based upon the indication of the handedness of the user. 26.The computer system of claim 23, wherein the processor is configured toreceive an indication of the orientation of the stylus with respect tothe display screen, and wherein the processor is configured to displaythe menu choices by displaying the icons in a menu positioned on thescreen based upon the indication of the stylus orientation.
 27. Thecomputer system of claim 23, wherein the processor is further configuredto: continue to display the icons after selection of an icon, receive anidentification by the user of a second object, receive a secondselection of an icon, and perform the action identified by the secondselected icon.
 28. The computer system of claim 27, wherein theprocessor is further configured to move the displayed icons across thedisplay screen in response to a corresponding movement of the styluswith respect to the display screen.
 29. The computer system of claim 23,wherein the processor is further configured to display, in response to auser selection of an initial icon, a second level menu of additionalicons, wherein the processor is configured to receive a user selectionof an icon with the stylus by receiving a selection of an icon in thesecond level menu, and wherein the processor is configured to performthe action identified by the selected icon by performing the actionidentified by the icon selected from the second level menu.
 30. Thecomputer system of claim 23, wherein the processor is further configuredto: display, in response to a user selection of an initial icon, asecond level menu of additional icons cascaded from the initial icon,receive a selection of an icon from the second level menu, and display,in response to selection of the icon from the second level menu, a thirdlevel menu of additional icons cascaded from the second level menu icon,wherein the second and third level menus are cascaded in substantialalignment from the initial icon, the processor is configured to receivea user selection of an icon with the stylus by receiving a selection ofan icon in the third level menu, and the processor is configured toperform the action identified by the selected icon by performing theaction identified by the icon selected from the third level menu.
 31. Ina computer system having a display screen, a graphical user interface,and a user interface selection device, a method of selecting actionsperformable with regard to an object having an associated representationon the screen, comprising: receiving a user request for a display ofmenu options corresponding to the actions performable with regard to theobject; displaying an initial context menu having a set of menu options;displaying successive cascaded context menus, the initial context menuand the successive cascaded context menus forming a cascade, whereineach context menu has a set of menu options, each context menu in thecascade remains at least partially visible each cascaded context menu isdisplayed upon selection of a menu option from a preceding menu in thecascade, and each cascaded context menu is displayed in close proximityto, but without obscuring, the option selected to cause display of thecascaded context menu; receiving a repeat selection of an option from acontext menu in the cascade; discontinuing, in response to the repeatselection, display of all context menus cascaded from the context menuin which the repeat selection was made.
 32. The method of claim 31,wherein: receiving a user request for a display of menu optionscomprises receiving a request initiated with a stylus user inputselection device, and receiving a repeat selection of an optioncomprises receiving a request initiated with a stylus user inputselection device.
 33. The method of claim 32, wherein each optionselected to cause display of a context menu in the cascade is visible.34. The method of claim 33, wherein the cascade comprises at least threecontext menus, wherein the repeat selection is received in anintermediate context menu, and further comprising: receiving, in theintermediate context menu, selection of an option distinct from therepeat selection option; and displaying one or more successive contextmenus cascaded from the intermediate context menu.
 35. The method ofclaim 33, further comprising: displaying, for each option that can beselected to display a cascaded context menu, an indicator; changing,upon selection of an option having an indicator, the appearance of theindicator.
 36. The method of claim 35, wherein a repeat selection ismade by selecting an indicator having a changed appearance.
 37. Themethod of claim 33, wherein each cascaded context menu appears below andsubstantially aligned with the menu option selected from the previouscontext menu in the cascade.
 38. A computer readable medium havingcomputer-executable instructions for performing steps comprising:receiving a user request for a display of menu options corresponding toactions performable with regard to an object, wherein the object has arepresentation upon a display screen of a computer system having agraphical user interface; displaying an initial context menu having aset of menu options; displaying successive cascaded context menus, theinitial context menu and the successive cascaded context menus forming acascade, wherein each context menu has a set of menu options, eachcontext menu in the cascade remains at least partially visible eachcascaded context menu is displayed upon selection of a menu option froma preceding menu in the cascade, and each cascaded context menu isdisplayed in close proximity to, but without obscuring, the optionselected to cause display of the cascaded context menu; receiving arepeat selection of an option from a context menu in the cascade;discontinuing, in response to the repeat selection, display of allcontext menus cascaded from the context menu in which the repeatselection was made.
 39. The computer-readable medium of claim 38,wherein: receiving a user request for a display of menu optionscomprises receiving a request initiated with a stylus user inputselection device, and receiving a repeat selection of an optioncomprises receiving a request initiated with a stylus user inputselection device.
 40. The computer-readable medium of claim 39, whereineach option selected to cause display of a context menu in the cascadeis visible.
 41. The method of claim 40, wherein the cascade comprises atleast three context menus, wherein the repeat selection is received inan intermediate context menu, and comprising further computer-executableinstructions for performing the steps of: receiving, in the intermediatecontext menu, selection of an option distinct from the repeat selectionoption; and displaying one or more successive context menus cascadedfrom the intermediate context menu.
 42. The method of claim 40,comprising further computer-executable instructions for performing thesteps of: displaying, for each option that can be selected to display acascaded context menu, an indicator; changing, upon selection of anoption having an indicator, the appearance of the indicator.
 43. Themethod of claim 42, wherein a repeat selection is made by selecting anindicator having a changed appearance.
 44. The method of claim 40,wherein each cascaded context menu appears below and substantiallyaligned with the menu option selected from the previous context menu inthe cascade.